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Abstract 

The  detection  of  low  signature  or  camouflaged  targets  in  cluttered  backgrounds  is  a  crucial 
problem  in  tactical  reconnaissance.  In  the  past  few  years,  imaging  spectral  and  polarimetric 
sensors  have  been  evaluated  for  this  application.  Although  these  sensors  have  separately 
generated  promising  results,  each  imaging  modality  alone  appears  to  have  not  achieved  the 
desired  level  of  target  detection.  Fusion  of  data  from  multiple  sensing  modalities  may  potentially 
improve  performance  to  acceptable  levels.  In  the  case  of,  a  key  issue  is  the  correlation  of  the 
spatial  location  of  the  false  detection  within  spectral  and  polarimetric  imaging.  This  paper 
presents  a  study  of  a  data  set  consisting  of  near  simultaneous  spectral  and  polarimetric  images 
recorded  from  sensors  colocated  on  North  Oscura  Peak  in  the  White  Sands  test  range.  The 
sensors  overlooked  a  scene  composed  of  natural  background,  military  vehicles,  and  camouflage 
material.  The  sensors  operated  in  the  visible  band  with  nearly  equal,  simultaneous  field  of  view. 
The  RX  anomaly  detection  algorithm  was  separately  applied  to  each  data  set  to  obtain  a  two 
dimensional  map  of  target  and  false  detections.  The  paper  will  analyze  the  correlation  of  false 
detections  for  image  fusion.  Background  segmentation  of  the  hyperspectral  and  polarization  data 
sets  was  also  examined. 

Introduction 

Target  detection  amid  natural  background  clutter  can  be  achieved  through  collection  and 
analysis  of  data  derived  from  optical  multispectral  or  hyperspectral  sensors.  The  reflection  or 
emission  spectral  signatures  depend  on  the  elemental  composition  of  objects  residing  within  the 
scene.  Targets  whose  elemental  composition  deviates  substantially  from  the  backgrounds  may  be 
more  easily  detected  using  radiation  wavelength  selective  detectors.  Although  the  schemes 
applied  to  the  optical  spectral  data  are  highly  selective  in  discriminating  targets,  it  is  necessary  to 
vastly  improve  the  sensitivity.  Detection  of  an  additional  feature  of  suspected  targets  has  been 
postulated  to  help  in  discriminating  objects  within  a  given  scene.  Radiation  polarization  is 
sensitive  to  surface  features  such  as  relative  smoothness  or  roughness1.  It  has  been  suggested  that 
detection  of  radiation  polarization,  in  combination  with  radiation  wavelength  sensitivity  maybe 
enhance  target  detection.  The  hyperspectral  data  were  recorded  to  enable  a  preliminary 
assessment  of  the  utility  of  polarimetric  /  hyperspectral  data  fusion.  Mathematical  fusion  of  data 
sets  has  been  examined  to  enhance  target  detection  using  a  different  imager2. 
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A  field  collection  of  polarimeteric  image  data  was  conducted  during  the  period  12-16 
October  1998  at  North  Oscura  Peak,  White  Sands  Missile  Range,  NM.  The  collection  was 
coordinated  by  Nichols  Research  Corporation  (NRC)  and  ERIM  International.  A  Naval 
Research  Laboratory  (NRL)  team  recorded  companion  hyperspectral  data  using  the  Dark  Horse  1 
(DH1)  sensor.  A  University  of  Alabama,  Huntsville  (UAH)  visible  spectrum  polarimetric  camera 
was  operated  by  the  NRC  team. 

This  report  concentrates  on  the  data  fusion  analysis.  Since  the  hyperspectral  data  was 
visible  spectrum  only,  visible  spectrum  polarimetric  data  was  the  logical  choice  for  fusion  with 
the  hyperspectral  data.  The  UAH  visible  polarimetric  camera  was  a  good  match  in  IFOV  to  the 
hyperspectral  sensor.  For  this  reason  the  UAH  data  set  was  chosen  for  detailed  analysis. 

In  order  to  provide  a  context  for  the  fusion  analysis  itself,  an  overview  of  the 
characteristics  of  the  Dark  Horse  3  and  UAH  sensors  is  provided.  The  various  steps  in  the  data 
reduction  chain  are  also  outlined. 

The  final  sections  of  the  report  summarize  the  results  of  the  fusion  assessment.  It  is 
found  that  data  fusion  based  on  “AND”  processing  provides  the  best  results.  Approximately  one 
order  of  magnitude  improvement  in  false  alarm  reduction  appears  to  be  possible  from  fusion  of 
the  two  data  sets. 

Sensors 


Several  sensors  were  employed  during  the  field  test.  NRC  provided  separate  MWIR  and 
LWIR  imaging  polarimeters,  ERIM  supplied  a  second  MWIR  imaging  polarimeter  and  UAH 
provided  a  visible  imaging  polarimeter.  NRL  provided  the  DH1  visible  hyperspectral  sensor 
outfitted  with  a  computer-controlled  scanning  table.  Detailed  descriptions  of  the  polarimetric 
sensors  are  available  in  separate  reports  from  NRC  and  ERIM.  Some  information  on  the  DH1 
sensor  is  provided  here  for  easy  reference.  More  detailed  information  is  available  from  NRL. 
Since  the  polarimetric  data  of  interest  in  this  report  was  collected  by  the  UAH  instrument,  a  brief 
description  of  this  instrument  is  contained  below  in  the  polarimetric  analysis  section.  More 
information  is  available  from  UAH. 

Hyperspectral  Sensor  (Dark  HORSE) 

The  Dark  HORSE  hyperspectral  imaging  sensor3  consists  of  a  standard  1 "  CCTV  lens 
(Navitar,  DO-5018),  an  f/2  imaging  spectrograph  (Instruments  S.A.,  CP  140)  and  a  high- frame- 
rate  CCD  camera  (Sarnoff,  VCCD512).  Imagery  collected  via  the  50mm  focal  length  lens  is 
dispersed  using  a  high  throughput  f/2  imaging  spectrograph.  The  spectrograph  employs  an 
aberration-corrected  concave  holographic  grating,  providing  a  flat  field  spectral  range  from  400 
to  850  nm.  A  25  jam  slit  width  was  used  for  all  data  collections.  However,  it  should  be  noted  that 
the  size  of  the  binned  CCD  pixels  (not  the  spectrograph  slit  width)  defines  the  sensor’s  maximum 
spectral  resolution  of  7nm. 


Digital  collection  of  the  hyperspectral  imagery  is  achieved  using  a  custom  high- frame- 
rate  CCD  camera.  The  camera  is  a  16-port  split  frame  transfer  CCD  with  12-bit  digitizers 
operating  at  a  maximum  frame  rate  of  200  Hz.  A  custom  high-frame-rate  interface  box  is  used  to 
merge  the  16  12-bit  digital  camera  output  channels  into  a  single  32-bit  channel  that  is  then  read 
via  a  digital  frame  grabber  board  (MuTech,  MV- 1 100).  The  512x512  silicon  focal  plane  array 
(FPA)  is  capable  of  capturing  digital  hyperspectral  data  cubes  with  a  maximum  of  512  cross-track 
spatial  pixels  and  512  wavelength  bands.  During  this  field  test  the  spectral  pixels  were  binned  via 
hardware  to  provide  64  wavelength  bands.  Spectral  pixel  binning  provided  adequate  signal-to- 
noise  levels  at  the  relatively  short  integration  times  (0.1  seconds  per  frame)  used  for  this  collect. 
The  instantaneous  field  of  view  (IFOV)  is  0.22  mrad. 

The  sensor  is  controlled  using  a  266  MHz  Pentium  II  PC  and  a  custom  software 
application  run  under  the  Microsoft  Windows  NT  operating  system.  The  user  interface  provides 
the  user  a  means  of  entering  required  input  parameters,  such  as  archival  file  names,  desired 
binning  parameters  and  camera  frame  rates.  In  addition  to  controlling  the  sensor  the  controller 
computer  is  used  to  unscramble  the  frame  grabber’s  32-bit  image  output. 

The  sensor  operates  in  a  pushbroom  mode  by  forming  a  spectrally  dispersed  image  of  the 
entrance  slit  onto  the  two-dimensional  FPA.  One  dimension  gives  spectral  information  and  the 
other  dimension  gives  spatial  information  in  the  cross-track  direction.  Scanning  the  slit  across  the 
scene  of  interest  gives  spatial  information  in  the  down-track  direction  and  results  in  a  three- 
dimensional  hyperspectral  data  cube.  Typically,  the  required  scanning  motion  is  achieved  by 
mounting  the  sensor  to  a  moving  airborne  platform.  For  this  data  collection,  motion  in  the  down- 
track  direction  was  achieved  via  a  computer-controlled  azimuth-elevation  scanning  mount.  For 
all  data  collected  the  sensor  was  scanned  at  a  rate  of  0.125  degrees/sec  while  holding  the 
elevation  angle  constant. 

Sensor  and  Target  Sites 

All  data  were  collected  at  North  Oscura  Peak,  White  Sands  Missile  Range,  NM.  All  sensors  were 
located  at  the  top  of  North  Oscura  Peak  at  the  “Atom  Site”.  Sensors  were  placed  on  a  concrete 
platform  (previous  page)  that 
overlooked  the  “Target  Site” 

(Figure  1).  The  “Target  Site” 
was  at  the  base  of  North 
Oscura  Peak  at  a  depression 
angle  of  approximately  30 
degrees  and  a  distance  of 
approximately  1500  meters 
from  the  sensor  platform. 

Targets 

A  variety  of  targets 
were  examined  during  the 
course  of  the  field  test, 
ranging  from  military 
vehicles  to  simple  geometric 
shapes.  For  the  purposes  of 


this  study,  The  Camouflage,  Concealment  and  Deception  (CC&D)  related  targets  are  the  focus  of 
this  study.  These  targets  included  military  vehicles,  vehicle  decoys,  and  Chemical  Agent 
Resistant  Coating  (CARC)  panels.  The  targets  were  positioned  along  a  jeep  trail  and  arranged 
such  that  the  military  vehicles  were  in  pairs  (Figure  2),  with  one  vehicle  obstructed  (via  natural  or 
manmade  cover)  and  one  in  clear  view. 


Figure  2.  Enlarged  picture  of  targets  (indicated  with  lines). 


General  Observations 

The  DH1  sensor  and  system  worked  well  over  the  course  of  the  field  test  with  no  significant 
hardware  problems.  Poor  field  conditions  (high  winds,  dust  storms  and  lack  of  power)  during  the 
later  portion  of  the  week  limited  data  collection  to  the  first  two  days  of  the  field  test.  Despite  this 
limitation,  a  significant  amount  of  data  was  collected.  Most  image  scans  covered  the  entire  target 
region  and  a  considerable  amount  of  ground  adjacent  to  the  targets.  Image  scans  were  also 
collected  both  above  and  below  the  target  region,  with  only  a  subset  of  the  targets  within  the 
sensor’s  field  of  view. 

In  addition  to  the  hyperspectral  data,  simultaneous  video  imagery  was  collected.  Additionally, 
digital  photos  of  all  of  the  CC&D  targets  were  acquired  and  a  map  of  the  targets’  relative 
positions  was  drawn  (see  above).  A  detailed  listing  of  the  targets’  exact  geolocations  with 
respect  to  the  sensor  is  available  through  NRC  and  ERIM. 


Wind  was  a  constant  concern  during  the  course  of  the  field  test.  Mild  winds  caused  the 
scanning  mount  to  sway  during  data  collections,  high  winds  made  it  necessary  to  abort  data 
collections  completely.  During  mild  winds  it  was  found  that  frame  rates  of  10  Hz  provided  the 
best  results.  At  this  rate,  data  were  sampled  rapidly  enough  to  minimize  image  pixel  smearing  yet 
slow  enough  to  avoid  the  use  of  high  scan  velocities.  Scan  rates  higher  than  10  Hz  did  not 
produce  reliable  data. 

Spatial  Correction 

The  spectrometer  optical  configuration  which  directs  radiation  to  the  focal  planar  array  is  not 
perfectly  ideal.  To  correct  spatial  aberrations,  the  spectrometer  is  calibrated  by  exposing  it  to 
mercury  and  krypton  lamps  placed  at  fixed  positions.  Software  corrects  each  pixel  within  the  two 
dimensional  array  to  generate  a  linear  spatial  response.  This  correction  reduced  the  observed 
aberration  from  as  much  as  10  pixels  in  the  outer  regions  of  the  focal  plane  array  down  to  1  pixel. 
Central  portions  of  the  array  displayed  much  smaller  spatial  distortions. 

Flat-fielding 

Radiometric  corrections  to  the  raw  data  are  also  required  because  the  spectrometer  radiation 
transport  efficiency  depends  on  wavelength.  Flat-fielding  procedures  were  used  to  correct  for 
the  wavelength  dependent  responsivity.  This  software  correction  was  applied  to  each  pixel  within 
the  image  following  exposure  to  a  spatially  uniform  radiation  source  of  known  intensity.  The 
figures  show  profiles  before  and  after  flat-fielding  corrections.  After  flat- fielding  corrections 
were  applied  to  the  data  editing  limited  the  hyperspectral  image  set  to  data  residing  between  the 
black  vertical  lines. 

Destriping 

The  visible  hyperspectral  image  contains  “streak”  artifacts  (Figure  3  a).  Most  of  the  streaks 
appear  as  lines  that  run  along  the  in-track  (horizontal)  direction.  These  streaks  are  attributed  to 
unstable  electronics  used  to  process  signals  in  the  CCD  array.  Additional  instability  is  attributed 
to  the  variable,  random  movement  of  the  Dark  HORSE  1  spectrometer  under  windy  conditions  at 
the  test  site. 


Figure  3a.  Horizontal  stripes  displayed  in  one  band  within  visible  hyperspectral  data  cube.prior  to 
“destriping” 


The  streaks  ar  e  particularly 
deleterious  for  anomaly 
detection  algorithms.  The 
presence  of  regions  showing 
sharp  gradients  will  generate 
false  anomalies.  Figure  3b 
depicts  the  vertical  profile 
(after  taking  the  median  in  the 
horizontal  direction)  in  bands 
10,  20,  30. 


Figure  3b  depicts  the  vertical  profile  (after  taking  the  median  in 
the  horizontal  direction)  in  bands  10,  20,  30. 


To  help  enhance  the  search  for  anomalies,  the  streaking  was  ameliorated  by  post  processing  of  the 
image  set.  Custom  software  reduced  the  apparent  streaking  by  a  complex  procedure  applied  to 
each  spectral  band  image.  The  method  involved  computation  of  the  row  medians,  determination 
of  the  row  offsets  needed  to  produce  the  smoothed  median  profiles  shown  in  the  figure  below, 
and  finally  correction  of  all  the  pixel  values  by  these  offsets.  Following  application  of  the 
destriping  algorithm,  the  horizontal  streaking  was  substantially  reduced  as  seen  in  the  image 
Figure  4a  and  the  profiles  shown  in  Figure  4b. 


Figure  4a.  Prominent  horizontal  stripes  removed  from  band  displayed  in  Figure  3a. 


Figure  4b  depicts  the 
smoothed  vertical 
profile  (after  taking 
the  median  in  the 
horizontal  direction) 
in  bands  10,  20,  30. 


Additional  Editing  of  the  Hyperspectral  Data 

Following  flat  fielding  and  spatial  distortion  corrections,  the  hyperspectral  image  data  set 
required  further  editing.  Although  the  calibrations  corrected  most  distortions,  image  editing  was 
still  required  to  provide  a  valid  image  set.  Specifically,  the  image  set  was  reduced  from  512  pixels 
in  the  cross  track  direction  down  to  the  central  256  pixels.  Additionally,  spectral  bands  1-6  and 
50-64  were  eliminated  due  to  remaining  distortions.  The  remaining  bands  (7-49),  correspond  to 
the  spectral  range  820-500  nm. 

Polarimetric  Analysis 


Stokes  images  can  be  used  to  characterize  the  degree  of  linear  and  circular  polarization  at  each 
individual  pixel  within  a  digital  image  scene  4,5 .  The  individual  Stokes  parameters  are  S0  (total 
intensity),  Si  and  S2  (linear  polarization),  and  S3  (circular  polarization).  The  University  of 
Alabama,  Huntsville  polarimetric  camera  was  used  to  collect  the  polarimetric  data.  The  key 
elements  of  this  camera  are  a  320  x  192  element  CCD  array  positioned  behind  a  rotating  retarder 
plate  /  fixed  polarizer  combination.  The  IFOV  is  .23  x  .26  mrad.  In  the  field,  the  retarder  was 
rotated  to  12  angular  positions  separated  by  intervals  of  15  degrees  and  an  image,  I(n),  was 
recorded  for  each  position.  The  12  images  were  spatially  registered  to  each  other  using  custom 
software  based  on  tiepoints  and  polynomial  fitting.  The  12  images  are  related  to  the  Stokes 
images  and  degree  of  linear  polarization  (DOLP)  in  the  following  manner. 
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The  Stokes  images  were  computed  by  inverting  this  equation.  Imperfect  registration  generated 
some  artifacts  in  the  Stokes  images  (especially  S3).  Improvements  in  registration  would  be 
desirable.  Shown  below  (Figure  5)are  processed  Stokes  images  derived  from  the  UAH 
polarimeter.  The  Si  and  S2  images  (Figure  5b,  5c)  suppress  clutter  significantly  while  permitting 
observation  of  some  of  the  targets  in  the  scene.  A  number  of  targets  were  not  detected. 


Figure  5a.  S0  Stokes  image 


Figure  5b.  Si  Stokes  image 


Figure  5c.  S2  Stokes  image  Figure  5d.  S3  Stokes  image 


Registration  of  Hyperspectral  Images  with  Polarimetric  Images 


The  hyperspectral  images  are  generally  similar  to  the  polarimetric  images.  However,  the  overlap 
between  the  two  sets  of  images  is  not  perfect  and  consequently  some  manipulation  of  the  image 
set  was  required.  We  adopted  the  strategy  of  warping  the  hyperspectral  images  to  the 
polarimetric  images.  This  involved  translation,  rotation,  magnification,  and  resampling  of  the 
hyperspectral  images  to  achieve  a  1  to  1  correspondence  between  hyperspectral  pixels  and 
polarimetric  pixels.  Part  of  this  process  is  spatial  co-registration  of  the  two  sets  of  images.  To 
register  the  data,  common  points  called  ground  control  points  (GCP)  were  identified  within  each 
image  set.  These  reference  points  were  used  to  connect  the  two  sets  together.  A  total  of  7  ground 
control  points  were  employed.  Since  our  linear  registration  method  required  four  GCPs,  three 
remained  to  assess  the  quality  of  the  registration.  Typically,  the  hyperspectral  and  polarimetric 
image  sets  were  registered  to  within  <0.8  pixels.  The  final  result  of  the  warping  process  was  an 
320  x  192  pixel  image  size  for  both  data  sets. 


RX  anomaly  detection 


Manmade  targets  located  within  natural  backgrounds  often  appear  as  spectrally  anomalous 
objects.  However,  the  detailed  spectral  signature  of  the  target  is  usually  not  available  a  priori. 
Anomaly  detection  algorithms,  such  as  the  RX  algorithm 6,  search  for  pixels  within  the  image  set 
exhibiting  a  spectrum  significantly  different  from  that  of  the  background.  The  RX  value  for  the 
pixel  at  a  row,  column  position  i,j  is 


RX(x  i,j)=(x  i,j-mean)T  *  (M1)  *  (x  ij-mean) 


where  x  ^  is  the  spectral  vector  at  position  i,j,  mean  refers  to  the  mean  band  value,  and  M"1  is  the 
inverse  of  the  covariance  matrix.  Figure  6  pictorially  outlines  the  RX  anomaly  search  process. 


.  Figure  6  pictorially  outlines  the  RX  anomaly  search  process. 


The  “three-dimensional”  image  cube 
(having  40  bands  in  the  hyperspectral 
image  set)  is  thus  reduced  to  a  two 
dimensional  RX  map  displaying  the 
degree  of  anomaly  within  the  image. 
Relatively  large  RX  values  are  to  be 
associated  with  detections  (either  true 
or  false).  Images  useful  for 
diagnostics  can  be  created  by 
thresholding  the  RX  maps  to  show 
the  pattern  of  detections.  A  further 
refinement  is  to  overlay  an  image 
mask  corresponding  to  the  known 
position  of  the  real  targets.  Maps 
created  in  this  way  from  the 
hyperspectral  data  and  for  the 
polarimetric  data  are  shown  in 
Figures  7a,  7b.  The  known  target 
overlays  are  in  color.  The  white 
areas  are  false  detections.  Raising  the 
threshold  decreases  the  number  of 
false  detections  at  the  expense  of 
missing  more  true  targets.  In 
generating  these  images  a 
preprocessing  step  called  principal 
component  analysis  was  used.  This 
statistical  tool  is  discussed  in  more 
detail  below.  Principal  components 
(PC)  2  through  7  were  used  for  the 
hyperspectral  data  and  PCs  2  and  3 
were  used  for  the  polarimetric  data. 


Figure  7a  Target  mask  superimposed  on  RX 
output  from  HSI  data. 


Figure  7b  Target  mask  superimposed  on  RX 
output  from  polarization  data. 


ROC  Curve 

Receiver  Operating  Characteristic  (ROC)  curves  are  used  to  visualize  a  classifier's  performance 
and  to  aid  in  selecting  the  proper  decision  threshold.  To  generate  a  ROC  curve,  a  series  of 
detection  results  for  different  threshold  values  are  compared  with  ground  truth  information  in 
order  to  determine  how  many  detections  are  true  and  how  many  false.  Each  pixel  within  the 
image  is  examined  to  determine  whether  or  not  the  RX  output  value  at  that  pixel  exceeds  a  given 
threshold  and,  if  so,  whether  the  pixel  resides  within  the  ground  truth  region  of  interest  (ROI) 
corresponding  to  one  of  the  targets.  The  ROC  curves  display  the  target  detection  probability 
versus  probability  of  false  detection.  Individual  target  ROIs  typically  contain  multiple  pixels.  In 


this  study,  a  target  was  considered  to  have  been  detected  if  any  pixel  within  a  target  ROI 
exceeded  a  given  threshold  value.  The  percentage  of  false  detections  is  defined  as  the  percentage 
of  pixels  lying  outside  all  target  ROIs  with  RX  value  exceeding  the  threshold  value. 

Principal  Component  analysis 

The  hyperspectral  data  set  has  40  bands  of  spectral  information.  From  a  statistical  point  of  view, 
this  corresponds  to  40  degrees  of  freedom.  Past  experience  has  shown  that  the  effective  number 
of  independent  degrees  of  freedom  is  far  less  -  typically  on  the  order  of  6  or  7.  Principal 
component  (PC)  analysis  is  a  convenient  way  to  reduce  the  data  set  to  a  dimension  of  this  order 
and  separate  image  information  from  noise.  The  method  works  by  forming  orthogonal  linear 
combinations  of  the  40  band  images  and  ranking  these  combination  images  by  variance.  PC  1 
(highest  variance)  is  usually  close  to  the  arithmetic  average  of  the  40  bands.  Since  it  is  nearly  an 
average,  it  is  dominated  by  background  clutter  and  is  usually  discarded.  The  PCs  with  the  lowest 
variance  (e.g  37th  ,  38th,  39th  and  40th)  are  dominated  by  noise  and  so  are  not  useful.  Best  results 
in  this  study  were  obtained  by  retaining  PC  2  through  7.  This  choice  has  several  advantages.  It 
rejects  a  good  deal  of  “ordinary”  clutter  (contained  in  PC  1).  It  reduces  remaining  streaks  and 
rejects  some  of  the  sensor  noise  (contained  in  PCs  8-40).  It  reduces  the  size  of  the  data  set 
thereby  easing  computational  requirements.  The  PC  algorithm  itself  is  not  computationally 
intense. 

The  same  idea  can  be  applied  to  the  polarimetric  data  set.  The  initial  data  set  is  4  dimensional  in 
this  case  (S0,  Si,  S2,  S3).  The  S0  component  was  rejected  because  it  is  clutter  dominated.  PC 
analysis  was  applied  to  the  remaining  3  components.  The  last  PC  is  nearly  equal  to  S3 
(mostly  noise  and  artifacts).  It  was  rejected  and  the  other  two  PC  were  retained. 

Fusion  of  Hyperspectral  Images  with  Polarimetric  Data  . 

Two  sets  of  independent  imaging  modalities,  visible  hyperspectral  and  polarimetry,  viewed  a 
common  set  of  targets  and  backgrounds.  These  two  image  sets  depict  common  targets  but 
differing  false  alarms.  Combining 
imaging  modalities,  or  fusing  the  data, 
was  investigated  as  a  possible  way  to 
enhance  target  detection  for  constant 
false  alarm  rates.  There  are  a  variety  of 
ways  to  fuse  registered  sets  of  images. 

Four  possibilities  are  examined  below. 

It  was  not  possible  to  use  data  recorded 
at  the  same  time  from  the  two  sensors 
for  this  exercise.  There  was  no  time 
when  the  UAH  polarimetric  camera 
and  the  DH1  spectral  sensor  were  both 
functioning  well.  Fortunately  the 
scene  did  not  change  very  much  during 
the  course  of  the  test.  The 
hyperspectral  scene  used  was  recorded 
on  Oct  13  at  1  pm.  The  polarimetric 
scene  used  was  recorded  on  Oct  12  at 
5:40  pm. 
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Figure  8a.  Schematic  showing  fusion  algorithm  using 
summation  of  RX  output  from  HSI  and  polarization. 


Sum  RX  Images 


In  this  approach,  the  RX  anomaly  algorithm  was  run  separately  on  the  hyperspectral  and 
polarimetric  data.  In  the  case  of  the  polarimetric  set,  only  Si,  S2?  and  S3  were  used  in  the  data  set 
because  S0  (total  intensity)  was  clearly  dominated  by  clutter.  An  “OR”  type  of  fusion  can  be 
implemented  simply  by  summing  the  RX  outputs  (Figure  8a).  This  is  referred  to  as  “OR” 
processing  because 
anomalies  in  either  data 
set  have  a  good  chance  of 
being  selected  as 
detections. 

shown  Figure  8b  is  the 
effective  ROC  curve  from 
summing  the  individual 
RX  outputs  of  the 
hyperspectral ,  and 
polarimetric  imaging  sets. 

Summing  the  RX  outputs 
(at  the  pixel  level)  of  the 
hyperspectral  (FI  SI)  and 
polarization  data  results  in 
a  modest  enhancement  in 
the  detection  of  targets  at 
constant  false  alarm  rate.  The  hyperspectral  set  was  taken  from  40  bands  (following  editing)  and 
selecting  the  principal  components  (PC)  2  through  7  (in  order  of  decreasing  eigenvalue  and 
variance).  The  polarimetric  set  was  derived  using  Stokes  images  Si,  S2,  S3  (S0  was  excluded  as 
being  clutter-dominated). 


ROC  Curve  (Sum) 


■HyperSpectr 

a 

Polarization 


Sum 


-6  -4  -2  0 

False  Alarm  Rate  (Log  10) 

Figure  8b.  ROC  curves  showing  fusion,  HSI,  and  polarization 


1 .  Multiply  RX  Images 


Observation  of  many 
uncorrelated  false  detections  in 
the  RX  output  from  the 
hyperspectral  and  polarimetric 
image  sets  suggests  another  way 
of  fusing  the  data  to  enhance 
target  detection.  Instead  of 
summing  the  data,  the  two  RX 
images  can  be  multiplied  at  the 
pixel  level  with  the  idea  of 
suppressing  false  detections  not 
present  in  both  RX  images.  This 
approach  is  sometimes  denoted 
as  the  “AND”  algorithm  because 
only  pixels  anomalous  in  both 
data  sets  are  likely  to  be  selected 
as  detections. 
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Figure  9a.  Schematic  showing  multiplication  of  HSI  and 
polarization  to  fuse  data. 


Figure  9b  is  the  effective  ROC 
curve  derived  by  multiplying  the 
individual  RX  outputs  of  the 
hyperspectral  (HSI),  and 
polarimetric  imaging  sets. 
Multiplying  the  outputs  (at  the 
pixel  level)  of  the  hyperspectral 
and  polarization  data  results  in 
greater  enhancement  in  the 
detection  of  targets  at  constant 
false  alarm  rates  than  does 
summing  the  image  sets. 
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3  Combine  Hyperspectral  and  Figure  9b  ROC  curve  from  multiplication  fusion,  HSI,  and 

Polarimetric  data  sets  and  polarization 


Single  Image  File 


perform  Anomaly 
Detection  on  large 
image  set 

A  third  approach 
(illustrated  by  Figure 
10a)  tried  to  combine 
the  hyperspectral  image 
set  with  the  polarimetric 
image  set  to  form  a  42 
dimensional  data  set 
(only  Si  and  S2  images 
were  used).  The  PC 
transformation  was  then 
performed,  and  only  PC 
2-7  retained.  The  RX 

algorithm  was  then  run  on  this  6  dimensional  data  set.  This  approach 
is  motivated  by  the  idea  that  the 
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Figure  10a.  Schematic  using 
image  cube  taken  from 
registered  HSI  and 
polarization  data  sets. 


large  data  set  might  be 
characterized  by  joint  Gaussian 
statistics. 

Figure  10b  shows  the  effective 
ROC  curve  from  performing  an 
RX  calculation  on  a  combined 
image  set.  The  large  image  set  is 
generated  by  combining  40 
bands  from  the  hyperspectral  set 
with  the  Si  and  S2  images  from 
the  polarization  set.  The  RX 
algorithm  was  subsequently 
applied  using  principal 
components  2  through  7  (in 


ROC  Curve  (Cube(Pol+HSI)) 


t- 

i 

3 

& 

0.8 

© 

u 

0.6 

Plh 

fi 

0.4 

#o 

C J 

0.2 

QJ 

+* 

0 

Q 

-6 


-4 


-2 


0 


■Hyperspectral 


Polarization 


Cube 

(Pol+HSI) 


False  Alarm  Rate  (Log  10) 

Figure  10b.  ROC  curve  from  registered  cube,  HSI,  and  polarization 


order  of  decreasing  eigenvalue  and  variance).  As  a  comparison,  the  hyperspectral  (HSI),  and 
polarimetric  RX  outputs  are  also  shown.  Performing  the  RX  on  the  combined  image  set  results  in 
no  significant  improvement  in  the  detection  of  targets  at  constant  false  alarm  rates. 


4.  Combine  Principal  Component  Analysis  of  Hyperspectral  data  with  Polarimetric  data 
and  perform  Anomaly  Detection  on  large  Image  Set 


The  fourth  approach  tried 
was  a  variant  of  the  third. 

The  PC  transformation  was 
applied  to  the  hyperspectral 
(HSI)  data  set  before 
combining  the  data  sets 
(Figure  11a).  The 
motivation  here  is  to 
optimize  the  hyperspectral 
data  set  before  fusion. 

Figure  lib  shows  the 
effective  ROC  curve 
obtained  in  this  way.  The 
large  image  set  is  generated  by 
combining  PCs  2  through  7 
from  the  hyperspectral  set  and 
Stokes  images 
Si  and  S2  from  the 
polarimetric  set.  The  RX 
algorithm  was  applied  using 
(PC)  2  through  7  (in  order  of 
decreasing  eigenvalue  and 
variance).  As  a  comparison, 
the  hyperspectral  (HSI), 
polarimetric  RX  outputs  are 
also  shown.  Performing  the 
RX  on  the  combined  image  set 
results  in  no  significant 
improvement  in  the  detection 
of  targets  at  constant  false 
alarm  rates. 


Figure  11a.  Schematic  showing  fusion  involving  PC  from  HSI 
forming  a  cube  with  polarization 
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Figure  lib.  ROC  curve  showing  fusion  from  PC 
reduced  HST  and  nnlarizatirm  HST  and  nnlnri7,pitirm 


Additional  insight  may  be  achieved  by  generating  a  scatterplot  of  the  registered  RX  outputs  from 
the  hyperspectral  imager  and  polarimetric  camera.  The  graph  plots  the  pixel  value  of  the 
registered  RX  output  from  the  hyperspectral  image  set  against  the  pixel  value  of  the  RX  output 
from  the  polarimetric  data.  It  can  be  shown  (figure  not  shown)  that  low  RX  values  from  the 
polarimetric  data  correlate  with  higher  RX  values  from  the  hyperspectral  imager.  Conversely, 
pixels  with  high  RX  values  from  the  polarization  data  correspond  with  low  RX  values  from  the 
hyperspectral  imager.  In  addition,  it  can  be  noted  that  pixels  showing  high  polarization  anomaly 
and  low  hyperspectral  anomaly  values  tend  to  appear  at  the  edges  of  targets  while  those  with  low 
polarization  anomaly  and  high  hyperspectral  anomaly  values  tend  to  appear  in  the  central  area. 


Used  in  this  way,  hyperspectral  and  polarized  images  may  complement  each  other  in  detecting 
targets  in  the  clutter. 


Preliminary  Fusion  of  Clustered  Image  Cubes 


This  paper  has  examined  searches  for  anomalies  with  in  the  background  clutter.  Clustering  of  the 
background  clutter  provides  another  approach  for  analyzing  the  data.  A  preliminary  examination 
of  clustering  the  clutter  is  summarized. 
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Prior  to  segmentation  of  the 
hyperspectral  visible  data, 
illumination  suppression7  was 
applied  to  all  pixels  within  the  image 
cube.  Figure  12  illustrates 
illumination  suppression  in  two 
dimensions.  The  first  principal 
component  direction  depicts  the  region 
of  highest  variance  and  presumably  runs  parallel  to  the  illumination  direction.  Surfaces 
orthogonal  to  first  principal  component  therefore  depict  regions  of  constant  illumination.  The 
surface  of  constant  illumination  was  chosen  to  intersect  the  mean  value.  Each  pixel  was  projected 
onto  a  hypersurface  using  the  virtual  shade  point  (minimum  pixel  value).  Prior  to  shade 
correction,  a  median  filter  was  applied  to  the  hyperspectral  image  cube  to  eliminate  singularities 
generated  in  the  projection. 


^  Projection 

Figure  12.  Illumination  equalization  method  illustrated. 


Background  clutter  clustering  is  achieved  by  using  the  unsupervised  classifer  k-means8.  The 
hyperspectral  visible  spectra  was  partitioned  into  six  clusters  plus  a  cluster  composed  of  pixels 
residing  beyond  the  fixed  distance  from  the  designated  cluster  means.  The  partitioned  regions 
correspond  to  targets  (Figure  13a).  Similarly,  the  SI  and  S2  components  from  the  registered 
visible  polarimetric  data  were  partitioned  into  two  components  plus  the  null  segment  (null 
segment  shown  in  Figure  13b). 

HSI,  6  Clusters  Polarization,  2  Clusters 


Figure  13.  Segmented  output  from  HSI  (a)  and  polarization  (b) 

Fusion  was  accomplished  by  multiplying  the  two  sets  of  registered  segmented  data  sets 
on  a  pixel  by  pixel  basis  (see  Figure  9b).  To  evaluate  the  potential  gain  from  this  version 
of  data  fusion,  ROC  curves  were  generated  from  the  previously  used  target  masks.  Again, 
the  false  alarms  in  the  data  sets  were  uncorrelated  and  some  enhancement  (reduction  of 


FAR  by  one  order 
of  magnitude  at 
60%  target 
detection  level)  in 
the  data  can  be 
achieved  using 
these  independent 
data  sets.  Other 
clustering  schemes, 
such  as  those 
testing  supervised 
classification 
algorithms  and 
number  of 
segments  are  being 
investigated. 
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Figure  14.  ROC  curve  comparing  multiplied  segmented  HSI  and 
polarization  and  segmented  HSI,  and  polarization. 


Summary  and  Conclusion 


Using  ground  control  points  to  register  the  hyperspectral  and  polarimetric  image  data,  the  data 
sets  were  combined  in  a  variety  of  ways.  The  best  ROC  curve  results  were  obtained  with  the 
“AND”  algorithm.  Thus,  it  appears  that  false  alarms  are  largely  uncorrelated  in  the  two  types  of 
imaging  modalities.  A  more  modest  ROC  curve  benefit  was  obtained  with  the  “OR”  algorithm. 
Combining  image  cubes  does  not  appear  to  enhance  the  detection  of  targets  at  constant  false 
alarm  rates.  Scatter  plots  reveal  that  polarimetric  RX  anomaly  imagery  tends  to  detect  the  edges 
of  targets  while  the  hyperspectral  methods  preferentially  select  the  central  regions.  This  suggests 
that  the  two  modalities  may  complement  each  other,  particularly  in  target  segmentation  or  image 
understanding. 


Confidence  in  these  results  is  limited  by  the  quality  of  the  hyperspectral  image  and  polarization 
data  sets.  Destriping  reduced  or  eliminated  spurious  streaks  in  the  hyperspectral  images,  but  there 
remained  a  number  of  artifacts  in  the  data  that  appeared  as  false  detections.  These  artifacts  also 
precluded  the  use  of  matched  filter  hyperspectral  detection  algorithms  since  these  methods  are 
particularly  susceptible  to  artifacts,  especially  in  combination  with  illumination  equalization.  This 
study  would  have  benefited  from  availability  of  better  quality  hyperspectral  data.  Benefit  also 
would  have  been  gained  from  better  internal  registration  of  the  raw  polarimetric  image  set. 
Suboptimal  registration  led  to  some  artifacts  in  the  polarimetric  data  set. 
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